What properties should an overall measure of test performance possess?

To the Editor: Obuchowski et al. (1 ) recently reviewed the uses and misuses of ROC curves and explained, in simple terms, some sophisticated solutions to frequent problems, thereby popularizing tools described elsewhere, notably in the book coauthored by Nancy Obuchowski (2 ). What prompts me to comment on this careful and timely review are two fundamental questions: What properties should an overall measure of test performance possess? How do we make better tests score better? ROC curves are indispensable and conceptually straightforward, but selecting a single overall performance measure requires some care. In referring to the “ROC curve and the measures of accuracy derived from it”, the introduction to the review acknowledges that there are several choices. However, like many similar texts, it jumps quickly to examination of the area under the ROC curve (AUROC) without pausing to ask whether the AUROC is the right, or the best, measure. This omission leaves parts of the mathematics without the necessary “philosophical” foundation. It may also promote the misconception that an AUROC calculation is the purpose of drawing a ROC curve. Now, by what criteria should performance measures be selected? The first criterion is that two tests that provide the same information should also score the same. However, if one reorders the test outcomes by, e.g., interchanging “atypical” and “equivocal” on the list from “normal” to “cancer”, the geometry, and therefore the area, will change; nevertheless, the clinical import of each test outcome and, hence, the clinical merits of the test remain unchanged. In fact, it is easy to provide an example [see page 31 of Ref. (2 )] in which a test classifies all patients correctly and yet has an AUROC of 0.5, suggesting complete lack of discrimination. A logical remedy is to assume the most favorable ordering when calculating the AUROC (3 ). Briefly, that means ordering the test outcomes by decreasing likelihood ratio, from ominous to reassuring. It means reassembling the segments of the ROC curve to make it concave. However, the review by Obuchowski et al. (1 ) does not address concavity issues. Insistence on concavity is not enough: Even when tests have concave ROC curves, their AUROCs may fail to rank them correctly. My oldest example (3 ) involves a clinical context where, by virtue of suitable symmetries, two tests are demonstrably equally useful yet have AUROCs as different as 0.85 and 0.90. Although the report giving this example is cited by Obuchowski et al. (1 ) in other contexts, their review does not address this potential shortcoming of the AUROC. Another compelling criterion is this: If a test can be emulated by superimposing pure noise, such as independent measurement error, on another test, it cannot be more informative than the latter and should never score higher; the same applies when test readings are binarized or otherwise coarsened. Once one acknowledges that the AUROC may mislead—which is my main point—it would be gratifying to be able to propose fail-safe ROC summary statistics. Unfortunately, there is a theoretical reason that any alternative statistics will have similar shortcomings: no performance measure based solely on the ROC curve and ignoring the pretest disease probability can rank competing tests without sometimes violating the expected utility principle of decision theory and, hence, clinical rationality (4 ). Performance measures must take pretest probabilities into account and be based on assessments of utility, i.e., clinical benefit and loss, or at least on a provisional utility model (pseudo-regret function) (3, 4). Tests are then ranked by the expected pretest–posttest difference in utility. For example, the simple quadratic (Brier) pseudo-regret function is J(r) 4r(1 r), where r is a disease (vs no disease) probability; J(r) 1 when r 0.5, i.e., when the diagnosis is maximally uncertain, and falls off toward 0 as r approaches 0 or 1, i.e., diagnostic certainty. The expected pseudo-regret drops from J(p), p being the pretest disease probability, to: